Add parquet_ans_processor #455
Conversation
| } | ||
| } | ||
|
|
||
| // Parse V1 ANS write set changes |
There was a problem hiding this comment.
We need to also parse V2 ANS write set changes
There was a problem hiding this comment.
why do we have to parse V2 ans wsc?
rust/processor/src/processors/parquet_processors/parquet_ans_processor.rs
Show resolved
Hide resolved
yuunlimm
force-pushed
the
yuunlimm/parquet-ans-processor
branch
from
67ec5f1 to
6e3a15e
Compare
| pub txn_version: i64, | ||
| pub write_set_change_index: i64, | ||
| pub registered_address: String, | ||
| pub token_standard: String, | ||
| pub domain: Option<String>, | ||
| pub subdomain: Option<String>, | ||
| pub token_name: Option<String>, | ||
| pub is_deleted: bool, | ||
| #[allocative(skip)] |
There was a problem hiding this comment.
@ying-w let me know if it's missing any fields
There was a problem hiding this comment.
missing expiration_timestamp and subdomain_expiration_policy
There was a problem hiding this comment.
@rtso do you know how to parse out the expiration timestamps?
There was a problem hiding this comment.
Primary names don't have expiration timestamp, only ans lookups do. This PR looks like it's missing moving ans_lookup_v2 to Parquet @yuunlimm
There was a problem hiding this comment.
There was a problem hiding this comment.
I also realized that this processor is still writing to the v1 ANS tables
all_current_ans_lookups,
all_ans_lookups,
all_current_ans_primary_names,
all_ans_primary_names,
We can include these in the deprecated tables and turn it off
There was a problem hiding this comment.
I don't think ans_lookup_v2 was a part of parquet migration.
There was a problem hiding this comment.
@ying-w do we need lookups in BQ?
There was a problem hiding this comment.
just realized this was for ans_primary_name_v2 not ans_lookup_v2
Yes we use both ans_lookup_v2 and current_ans_primary_name_v2 and current_ans_lookup_v2
if it's just removing ans_primary_name_v2 i think that is ok w/out moving to parquet but if going to remove current that would like the non-current to be moved to parquet
yuunlimm
force-pushed
the
yuunlimm/parquet-ans-processor
branch
from
6e3a15e to
10efc4b
Compare
| #[async_trait] | ||
| impl ProcessorTrait for ParquetAnsProcessor { | ||
| fn name(&self) -> &'static str { | ||
| ProcessorName::AnsProcessor.into() |
There was a problem hiding this comment.
do we want a processor name that can differentiate parquet and regular ones? I see this method is implemented in different ways: 1) from processor name 2) hard coding
There was a problem hiding this comment.
good catch! will fix this.
yuunlimm
force-pushed
the
yuunlimm/parquet-ans-processor
branch
from
76a0e6b to
96228f8
Compare
Description
Test Plan